Method of automatically verifying document content

ABSTRACT

A news and/or search engine system predicts events and stories based on existing events, stories, etc., extracted from a knowledge domain. These predictions can be used for a number of different purposes, including locating new content and actual event outcomes.

RELATED APPLICATION DATA

The present application claims priority to and is a continuation of Ser.No. 12/191,916, now U.S. Pat. No. ______ which latter application inturn claims the benefit under 35 U.S.C. 119(e) of the priority date ofProvisional Application Serial no. 60/955,775 filed Aug. 14, 2007 bothof which are hereby incorporated by reference. The application isfurther related to the following applications, all of which areincorporated by reference herein:

-   Temporal Document Sorter & Method; Ser. No. 12/191,809 (attorney    docket number 2008-1)-   Temporal Document Trainer & Method; Ser. No. 12/191,830 (attorney    docket number 2008-2)-   News Aggregator and Search Engine Using Temporal Decoding; Ser. No.    12/191,896 (attorney docket number 2008-3)-   Temporal Document Verifier & Method; Ser. No. 12/191,927 (attorney    docket number 2008-5)-   User Based Document Verifier & Method; Ser. No. 12/191,941 (attorney    docket number 2008-6)-   Event Based Document Sorter & Method; Ser. No. 12/191,151 (attorney    docket number 2008-7)-   Temporal Document Sorter & Method Using Semantic Decoding and    Prediction; Ser. No. 12/191,973 (attorney docket number 2008-8)-   Temporal Based Online Search & Advertising; Ser. No. 12/191,199    (attorney docket number 2008-9)

FIELD OF THE INVENTION

The present invention relates to electronic systems and methods fordetecting and differentiating document content, particularly in thetemporal and geographic domains. The invention has particularapplicability to news aggregators, search engines and other automatedsystems where is it desirable to predict stories, events or similarcontent related to a query or topic.

BACKGROUND

Internet based news aggregators are well-known in the art. An example ofa contemporary system is that provided by Google at its News site, showngenerally in FIG. 7. Google News automatically gathers stories from anassortment of news sources worldwide, and automatically arranges theminto a variety of categories/topics as shown in FIG. 7. Typicallyspeaking systems such as this are designed to present what they deem themost relevant stories within the interface shown in FIG. 7, usingautomated algorithms which measure human interest/relevance ofindividual news stories. This is done primarily by identifying a numberof factors, including determining the quality of the source of the news,page views, search queries and personal preferences as explained in USPublication Nos. 20050165743; and 20050060312; all of which areincorporated by reference herein.

Google News also automatically updates the topics and news stories on aperiodic basis. One limitation of such system, however, is that there isno (apparent) discrimination made by the Google news algorithm to sortthe stories in actual chronological order within the main news page. Forthis reason, as seen in FIG. 7, the main story highlighted for theGeorgia-VA Tech football game is entitled “Preview” and is dated some 18hours ago. In fact, the story beneath such highlighted entry is morerecent and gives the actual outcome of the contest: Georgia in fact hasalready won the game. Accordingly the Google News aggregator, whilecompiling relevant content, tends to accumulate a lot of stale contentwhich is not very timely but which nonetheless is prominently displayedbecause of the algorithm computes importance.

At the same time it should be noted that by selecting the entry in FIG.7 one can see a more comprehensive listing of stories from the newsaggregator, including a chronological sort of the same. However eventhis aspect of the aggregator has limitations, because while the newsstories are identified by their release time (i.e., 1 hour ago), thisparameter is not in fact helpful for identifying the actual temporalquality of the content of certain stories. This is because many newsagencies/sources release stories which merely duplicate content fromearlier stories, and with little or no new added content relevant to astory therein. These repeated stories can bear a recent time stamp andthus be pushed (incorrectly) to the top by the Google temporalizer tosuggest that they are very recent.

An example of the duplication of content can be seen in FIG. 8, in whichthe top four stories, as sorted by the Google chronologizer, actuallycontain identical content even though they were time stamped withdifferent recency values. This figure also shows that these four storiesactually duplicate content dealing with the governor of California whichwas already first extracted some hours earlier from the Salon newssource. FIG. 8 also depicts the problem noted above, namely, that thestory shown with the dashed arrow (from Monsters & Critics) actually hasnewer content not found in the identified most recent articles. This canbe confirmed from examining FIGS. 8A and 8B; the M&C article clearlyevidences additional recent content relating to the governor'shospitalization.

The effect is particularly pronounced during the time in which certainevents (or their reporting), such as sporting events, elections, naturalcatastrophes, accidents, are taking place. That is, the updating ofscores is something that tends to lag significantly behind otherstories. This makes it hard to review the news at a glance andimmediately identify the current state of the certain events.

The situation is exacerbated by overseas news bureaus which pick up USnews stories and then repeat them verbatim at a later time. For instancea sporting event may start at 5 p.m. PST in the US and end at 8 p.m. Thenews then is disseminated overseas, and then reported on by severalforeign sources during their respective days. So as a practical matter,at 11:00 a.m. PST the next day, the foreign news source storiesdescribing the kick-off the game (not the result) are just beingpublished fresh in their respective domains. From the perspective of theGoogle type algorithms, which only appear to examine explicit timereferences, the foreign stories describing the beginning of the gameappear more recent than stories describing the result. The result is anaggregation of content that is mismatched in time.

Moreover the same lack of temporal relevance problem also exists withsearch engines purporting to render relevant results to users. Whilesuch systems typically include some mechanism for selecting “recent”content, there is no mechanism available to ensure that such content isindeed fresh and not simply a repeat of older, stale material. A similarsituation can be found in the Blogsphere as well, where it is not easyto determine the actual temporal relevance of material.

An example of this problem is seen in FIGS. 9 and 9A of the prior art.Here a query made to “Sharks Hockey Score” late in the evening onDecember 28 reveals nothing useful in fact concerning the game which hasjust completed against their opponent of the evening: Phoenix. No matterhow the stories are sorted, by relevance or date, there is noinformation about the game which the subscriber can glean, even thoughthe game had concluded and at least one news source had reported thefinal score.

To get such information one must leave the news aggregator and visitanother site, a fact, of course, which is undesirable from theperspective of trying to maintain the user's attention on the newsaggregator. The problem is exacerbated with smaller computing devicesand cellphones as well, where display space is limited.

Accordingly there is clearly a long-felt need for a temporal-baseddocument sorter which is capable of addressing these deficiencies in theprior art.

SUMMARY OF THE INVENTION

An object of the present invention, therefore, is to overcome theaforementioned limitations of the prior art.

Accordingly one aspect of the invention concerns a system and method ofautomatically classifying temporal characteristics of electronicdocuments with a computing system.

Another aspect concerns automatically sorting electronic documents bytheir temporal characteristics.

A further aspect of the invention is directed to automatically trainingan electronic document sorter to classify documents.

Still other aspects of the invention concern comparing electronicdocuments to identify content differences, content matches, and temporaldifferences.

Another aspect of the invention is directed to automatically presentingelectronic documents in accordance with their temporal characteristicsto users, including on search engines, news aggregators, etc.

Yet other aspects of the invention concern processing search queries inaccordance with temporal characteristics of documents.

A further aspect of the invention concerns identifying events andlocales in news stories, to determine appropriate ordering and contentsources for stories.

Another aspect of the invention involves automatically verifyingtemporal values of electronic documents through additional third partysources, including human contributors.

Other aspects of the invention involve automatically identifying andusing human contributors for news/temporal content, includingdetermining optimum participants for such contributions. Related to thisaspect are interfaces, websites and other tools designed to facilitatecollection of ranking data from volunteers.

More specific aspects of the invention are concerned with collectingcontent for news stories, particularly sports, financial, election,disaster and other stories, and presenting such in a manner designed togive the most up to date status information for such events.

Another aspect of the invention is the use of semantic information tocontribute to the temporal decoding, sorting and presentation ofcontent.

A further aspect of the invention involves the classification of newsstories, and the prediction of related developments expected for suchstories.

Yet another aspect concerns automated advertising that relies upon anduses temporal information to enhance online ad auctions, advertisingplacement and search engine behavior.

These aspects of the invention (and others described herein) arepreferably implemented as one or more computer software routinesembodied in a tangible media and adapted to cause one or more computingsystems to perform the required operations.

It will be understood from the Detailed Description that the inventionscan be implemented in a multitude of different embodiments. Furthermore,it will be readily appreciated by skilled artisans that such differentembodiments will likely include only one or more of the aforementionedobjects of the present inventions. Thus, the absence of one or more ofsuch characteristics in any particular embodiment should not beconstrued as limiting the scope of the present inventions. Whiledescribed in the context of news aggregators, search engines, blogcompilers and related systems, it will be apparent to those skilled inthe art that the present teachings could be used in any Internet basedsystem in which it is desirable to identify, compile and presentdocuments based on a temporal ordering.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart illustrating the preferred steps performed by adocument temporalizing system and process implemented in accordance withpreferred embodiments of the present invention;

FIG. 2 illustrates a preferred basic sorting process performed byembodiments of the present invention to classify documents and sort themin a temporal sequence;

FIG. 3 illustrates a relationship between semantic tags and temporalinterpretations for a particular topic/category of documents as utilizedby embodiments of the present invention to classify documents and sortthem in a temporal sequence;

FIG. 4 illustrates the preferred steps performed by a document temporalanalysis and scoring system and process which can be used in conjunctionwith the aforementioned document temporalizer of FIG. 1 and otherrelated embodiments;

FIG. 4A illustrates one aspect of the invention which optimizes the useof local sources for news items;

FIGS. 4B, 4C and 4D illustrate examples of the preferred approach forparsing and analysis of temporal components of electronic documents inaccordance with teachings of the present invention;

FIG. 5 illustrates the preferred steps performed by a verificationprocess which can be used in conjunction with the aforementioneddocument temporalizer of FIG. 1 and other related embodiments;

FIG. 5A illustrates one aspect of the invention which optimizes the useof local sources for verifying temporal aspects of events;

FIG. 6 illustrates a ranking system used in conjunction with theaforementioned document temporalizer of FIG. 1 and other relatedembodiments;

FIGS. 7, 8, 8A, 8B, 8C, 9 and 9A depict content presented by anoperation of a prior art news aggregator and search engine;

FIG. 10 depicts operation of a search/advertising engine implemented inaccordance with teachings of the present invention.

DETAILED DESCRIPTION

The present invention is concerned with identifying temporal differencesbetween documents, which typically are manifested in the form of someform of content differences between documents. A “document” as usedherein is intended to be understood in the broadest sense to includehuman or machine perceivable materials in electronic form. Whiletext-based documents are described herein with respect to a preferredembodiment, it will be understood that other applications of theinvention could be employed in other domains to include audio and videoinformation. In such latter cases the invention can be used to operateon text data extracted from audio content (such as by transcriptions orvoice recognition) tags describing such multimedia files (auto generatedor contributed by human reviewers) or other metadata associated withsuch files that can be analyzed temporally.

The content differences may in turn be defined as either: 1) net contentadditions or deletions; and/or 2) content semantic variations relatingto an ongoing event or story suggesting a temporal change. These contentdifferences, while not perfect indicators of course of temporalqualities of documents, are nonetheless strong indicators which can beexploited to compile and sort large numbers of documents such asutilized at news aggregators, search engine indices, blog searchindices, RSS feeds, etc.

For example, assume a document #1 has a certain content C_(A)-designatedcollectively as a text string {A1, A2 . . . . Ax} where An representsindividual words. The latter make up individual sentences S1, S2, etc.As seen in FIG. 2, later reportings on such story typically augment suchcontent directly, to result in a combined content document #2 containingC_(A)+C_(B), where C_(B) is an additional text string {B1, B2, . . . By}of additional sentences. This additional text string may be simplyappended or embedded in different ways within the original content.

In any event, a person seeing a new document #2 with combined contentC_(A)+C_(B) would perceive such to be a superset of the content of theoriginal #1 and thus for purposes of the present discussion such can betreated as a later edition of the original. Stated another way, theexistence of new data/content in document #2 can be associated with thecreation of new information which occurred—most likely—at a later time.

Similarly, some reportings may intentionally abridge or be earlierversions of the original story and thus a document #3 may evince acombined content C_(A)-C_(B). Again a human observing document #3 withcombined content C_(A)-C_(B) would perceive such to be a subset of thecontent of the original #1 and thus for purposes of the presentdiscussion document #3 can be treated as an earlier edition of theoriginal.

Other documents, such as documents #4 and #5, may contain very similarcontent C_(A′) or C_(A″) to document #1 but differ semantically in amanner which is temporally significant and helps to identify a temporalorder. This is explained more fully below, but a quick example would bea story reporting on an ongoing sporting event. In such case the text ofthe story may be almost identical except for the difference in time ofthe contest, the score, and other similar contest related parameters.Thus if story #1 reported that the score of the Georgia-Virginiafootball game was 10:0, and story #4 reported the score as 10:3, it isnatural to classify document #4 as reflecting more recent information.The existence of other semantic variations can be used effectively tointerpret a temporal rating for documents containing similar contentdirected to a topic that is a future event, an ongoing event, or anongoing evaluation of a prior event.

Furthermore, certain types of stories can be divided logically intotemporal classifications based on their underlying nature. For example,a developing story with respect to a terrorist investigation may proceedroughly as follows:

1) Bomb alert issued;

2) plot uncovered;

3) suspects detained

4) suspects identified

5) plot details revealed, etc.

Since this is a very common sequence, it is clearly useful to be able todifferentiate and classify documents accurately with respect to thistype of logical order which translates into a temporal sequence. Ifduring a review of documents therefore it is found that document #1 isrelated to a bomb alert therefore, and document #4 updates such storywith more plot details relating to the alert, then it can be safelyassumed that the latter contains more recent information.

Similar examples can be found in other fields; for example, in thebusiness field, a number of people are keenly interested to know whetherthe Fed chairman has actually raised rates. Prior to such announcementthere are typically dozens of stories predicting such event. The abilityfor a system to cleanly and quickly identify the actual decision eventis extremely useful. The same is true with respect to company earningsand similar financial event reportings.

High profile court trials also present a similar logistical challenge,because a significant amount of press is created prior to theannouncement of the verdict. The latter is buried in a sea of noiseuntil enough people have read the result to make it relevant enough tothe news aggregator. By such time it is often no longer “news” in thecontemporary sense.

Finally some documents, such as document #6 may be effectively identicalduplicates of original document #1. This scenario is explained above inconnection with the prior art system. Both documents evince the sametemporal value with respect to a particular event, even if they were notboth created at the same time.

These scenarios define a rough temporal change scale shown in the bottomof FIG. 2. This is not intended to be exhaustive and it will beunderstood that other formulations of content comparisons could bedeveloped and placed along this temporal scale.

FIG. 1 depicts a process 100 employed by preferred embodiments of thepresent invention to identify and sort documents in a temporal order. Atemporal classifier training step 110 is performed based on a tuning set120 and documents 121. The temporal classifier is preferably anartificial intelligence software routine, such as a natural languageengine executing on a computing system.

The temporal classifier preferably can be configured in the form of aterm destination-matrix, in a manner typically used in so-calledvector-based call routing used in speech recognition/routing systems andrelated systems. These systems work by transcribing calls made by humansto live operators who then interpret the spoken utterances and theninterpret the caller's request by directing them to a specificdepartment, person, etc. The basic theory is that the system breaks downthe user calls into distinct groups of words that it then begins toassociate with individual destinations. By analyzing a sufficientlylarge number of samples eventually the system develops enough examplesto compile a term-destination matrix, which allows for dissecting newcalls and matching them, based on their content overlap, to priordecoded calls made to the system.

The same phenomenon is studied in search engines as well, in that userqueries are logged along with the results presented in a search list.The user's selection of entries from the search list is then alsoevaluated to develop correlations for later users' searches. When alater search is made against the same search terms by a different userthe search engine factors in some weighting for the results based on theprior observed behavior for the prior user.

This same principle can be applied in the present invention as well. Themain difference in the present application is that the terms are derivedfrom analyzing documents such as news stories or web pages instead ofcaller transcriptions. Determining the “category” of a new document is arelatively straightforward exercise well-known in the art, and theaforementioned vector based approach would be one option. Othertechniques will be apparent to those skilled in the art. Thedestinations in this instance are the temporal classifications, so thatthe documents are sorted effectively into individual bins representing adistinct temporal interpretation value for individual categories.

Thus for a set of categories (C1, C2, . . . Cn) the natural languageengine is trained by presenting a set of documents relating to theindividual categories. For example, documents D1a may pertain to sports,particularly hockey events. D2a may pertain to business, particularlycompany earnings events, and so on. These documents are collectedthrough any convenient mechanism and developed into tuning sets 120which are used to teach the classifier how to interpret documents from atemporal perspective. The tuning sets are preferably developed byculling stories pertaining to the particular topics, and compilingexamples of different distinct temporal characteristics. For example, inthe aforementioned situation of hockey sporting events, the tuning setmay contain a set of K distinct documents representing K distincttemporal values in chronological order, ranging from an oldest to a mostrecent story pertaining to a hockey sporting event. The content of the Kstories is again preferably selected to accurately convey the temporallevel/value to be associated with the story. For instance, a first(oldest) story may contain content directed to a “preview” of the“upcoming match” at a particular date/time. A second story may commenton the expected team lineups, last minute scratches of players, andcurrent rink conditions. A third story may describe the score at the endof the first period of play, along with shots on goal, penalties, etc. Afourth story may describe the final score of the game with completestats. A fifth story may provide a re-cap of the game, along with quotesfrom the players, coaches, etc. These are but examples of course, and itwill be understood that the temporal classifications may be more coarseor granular depending on the particular field of interest.

The temporal classifications can be set up in advance or determinedautomatically from analyzing large collections of documents. They can beused, for example, to create rough cuts/divisions of other sets ofdocuments whose temporal qualities are not known ab initio. Furthermorethey can assist in determining an approximate initial temporal value fordocuments of particular categories.

Then the documents are sorted/annotated at step 130 by a combination ofboth human and machine logic if desired. The benefit of employing ahuman operator during this training step is that they can more easilyresolve ambiguities in temporal order. The sorted/annotated referencedocument sets (D1, D2, D3 . . . Dk) represent large collections ofdocuments which can be used later as a reference or benchmark to helpinterpret and classify the temporal qualities of a new document. Thesorting can also be based on automated observations of humans reviewingdocuments having different temporal order, to determine a sequence thatsuch persons used in reviewing such materials. Since most people areexpected (or can be easily trained) to review content in chronologicalsequence, this can be another source of temporal reference.

At step 140 information about the individual collections (D1, D2, etc.)can be stored in a table such as the following:

Category N Term Temporal Documents Vector Keywords Interpretation Pa1,Pa2, Pa3 . . . Pan Vfirst Wa1, Wa2, Wa3 . . . T_(first) Pb1, Pb2 . . .V1 Wb1, Wb2 . . . T₁ . . . . . . . . . Plast1, Plast2 . . . VlastWlast1, Wlast2 . . . T_(last)

In other words, the natural language engine can process the documentswhich are correlated to a particular temporal interpretation to extractkeywords which best represent or signify the presence of a documentwithin such temporal order. For example in the context of a sportscategory for hockey, the keywords for different temporal interpretationsmay include the terms discussed above, such as {(team name), preview,upcoming, face-off, start-time, expected team lineups, injury scratches,rink conditions, shots on goal, penalties, losing, lost, winning, won,secured a victory, first period, second period, third period, finalscore . . . } Linguistically speaking the pairings may consist ofsubject/predicate pairs. A number of semantic variants will of course beincluded as well for such words/phrases. A combined term vectorrepresenting each respective row of interpreted documents may also becompiled for later reference. Again while shown in the context of asports document analyzer, the same principles could be extended and usedwith any type of content to be analyzed for temporal qualities.

Examples of keyword tagging and temporal interpretation are shown inFIG. 3. For each document category or topic, a set of keyword/phrasetags is developed, either explicitly or as part of a documentdecomposition process discussed above. These keyword tags are thenmapped into a spectrum of temporal interpretations T0 through TF. Itwill be apparent to those skilled in the art that these are butexamples, and that other tags could be used instead in these topics.Moreover, the identity of the keyword tags will obviously vary fromtopic to topic. It is possible that the content tags could also becomprised of other types of data, including images/graphics which can becharacterized by suitable metadata that can function as content tags.Other multi-media data, for example audio can also be used since it canbe converted into text as well.

In some instances of course the keywords may be dynamically andnumerically linked to variables which change in predictable order. Forexample, a keyword/phrase may be in the form of “score is xxx-yyy” wherexxx and yyy are variables which change with time according to an eventscore. This same concept may be expressed in many different ways ofcourse (“home team is leading by a score of xxx to yyy” and the like).

Thus scores, event timers and related data may also be keywords. In thecontext of an election event, for example, the terms “xxx % ofprecincts” or “yyy votes” may represent keywords which changes and whichare to be monitored, so that stories can be differentiated temporallysimply by examining the respective % of precincts identified in thestory, or by comparing the total votes counted. In this fashion a storywhich reports on a higher number of precincts or higher vote tally canbe bumped ahead of earlier stories with smaller respective figures.Again similar concepts could be used in other fields where numericvariables (or semantic relatives) help to denote the progress of anevent, such as when there are figures associated with event, as in thenumber of sets in Tennis, incident rates, the number of casualties in anaccident/catastrophe, a number of products bought/sold, entertainment(movie/music) audience/box office figures, a number of shares traded,the date, a health condition, and other quantifiable physical variables(such inches of rain/snow and others which will be apparent to thoseskilled in the art).

In FIG. 1 at step 150 the system begins collecting a reference or seedset of new documents relating to one or more categories. Again, this canbe done using any known technique, including the prior art algorithmsnoted above for the Google news compiler. The raw material for suchstories can be extracted from a number of sources, including from searchengines 151, blogs 152, other content aggregators 153 and miscellaneoussources 154, which could be message boards, RSS feeds, etc. As notedearlier in some applications the source could be text data derived fromaudio/graphics/video based files, including audio transcriptions, speechrecognized data, or other metadata for such multimedia files.

The reference documents could also be sourced from a web page loaded ina user's browser. That is, while a person is reviewing an onlinedocument, the contents of the page (as well as other user interactionswith the content, such as viewing, highlighting, etc) could bedynamically captured and sent as source material for the presentinvention. Other aspects of the person's session—such as a search querywhich triggered the page view and review—could be sent along as well.The search query could be analyzed to determine an appropriatetopic/event or subject/predicate to be examined for temporal qualities.Based on this information the invention, as noted below, could return aset of results that are presented to the online viewer, preferably whilethey are still reviewing the electronic document (or related documentsfrom a search). In this fashion the invention can instantly anddynamically inform a web surfer of more recent content dealing with thesubject of the page.

The reference or seed set of new documents could also be based on spokenutterances which are recognized by a speech recognizer and thenconverted into text, or from SMS based text messages, etc. Theutterances could be provided by users calling in to report on events asthey happen, which can be exploited (as noted below) to gain morecurrent information concerning localized events such as disaster,accidents, etc. The identity or calling region of a caller can bedetermined in any number of ways, including conventional caller IDmechanisms, ANI techniques, etc. This can be used to control/filter aset of incoming speech related reports, so that electronic documentsbased on persons reporting closer to the scene of the event receivepriority in decoding (recognition) and temporal sorting.

The documents are then sorted by category by any convenient mechanismagain as shown in step 160, to form categorized sets SD1, SD2 . . . SDM,etc. These can then be sorted by a scoring step 170 to determine atemporal order. This is also useful for determining the relative stateof development of a new event which was not reported on before so thereis no earlier known content.

Preferably the sorting is done in two different ways; a first sort isdone to compare each document to a reference document set as establishedfor the category in question as discussed above in connection with steps130 and 140. This comparison can be done in any number of ways,including by checking for an overlap in keywords, a vector similaritycomputation, etc. In the final result a calculated temporal score can beidentified for each new document based on a comparison to the priorreference sets. Based on this temporal score the documents are sortedand then placed into distinct temporal bins reflecting the temporaldistinctions defined for such category.

Other techniques which are well-known in the art can be used as well,and it should be apparent that other benchmark/reference sets could beused to compare the new seed documents. The only requirement is that thealgorithm be able to reasonably make a rough determination on whichgeneral temporal order a document should be classified into for aparticular category—i.e., in one of the ranges Tfirst through Tlast.This completes a coarse sort.

One apparent advantage of the above approach is that duplicates ofexisting documents can be quickly identified and filtered. The removalof duplicate news stories, for example, would improve the look of a newsaggregator significantly. It is conceivable for example that somethreshold of content differences could be established to require that adocument exceed before it is actually classified as a new member of thedocument set.

In any event after a coarse sort, any newly accepted documents withineach temporal sorting bin are again sorted on a more granular level tocompare them against each other. In this second scoring step, a naturallanguage engine could compare pairs of documents in sequence to see iftheir respective order is correct, or if it should be switched. Againthe interpretation intelligence used by the natural language engineshould be geared towards providing higher scores for documentscontaining content reflecting later stages of the event in question. Theprogramming for this can be achieved in any desired manner depending onthe category of content in question. For example in the context of ahockey game, a particular story containing the terms/phrases won, lost,secured a victory, was outshot, etc., would be rated higher than a storycontaining the terms winning, late in third period, etc. Similarexamples will be apparent to those skilled in the art for other types ofcontent.

To further enhance the scoring, confidence levels and thresholds may beintroduced at step 172. This can include additional factors such asexamining a timeliness score for a source of the content in question.For example, certain new sources may develop reputations for breakingstories ahead of other entities. Over time a reputation or trust scorecan be established for certain documents based on their origin. Thereputation/trust score can also be used to modify the temporal rankingof the document. This feature is elaborated and explained in more detailbelow in connection with FIGS. 4 and 5.

In other cases various forms of weightings may be employed at step 173,so that the presence of certain key terms is favored and scored morehighly than for other content. For example certain verb forms may behighly favored when they reflect past tense, as opposed to an ongoingsituation. These weightings can be used to modify the temporal rankingof documents.

At step 171, the system can then (optionally) output the result of a topN list (for example N could be 3, 5, 10, etc. depending on presentationlimitations) to report on what it perceives to be the most currentcontent on the topic in question. By including a reasonable number ofcandidates the likelihood is of course much higher that the most recentstory in fact will be presented more readily. The ranking data can thenbe verified again, through another machine verifier of some kind (whichcould be another group of natural language examiners) or by humanobservation. The latter may be simpler and relatively easy to do by atrained team if the number of topics is not too large, and has theadvantage again of being more accurate. The ordering could then beconfirmed or modified as needed to create the initial ranked set ofdocuments. Alternative embodiments of the present invention may employ aseparate webpage or website whose members are allowed to rate thetemporalness of documents, as seen and discussed in detail below inconnection with FIG. 6. This has the advantage of potential increasedaccuracy as well as being attractive to certain types of Internet users.

Returning to FIG. 1 at step 180 the ranked set can be presented asdesired to persons viewing the aggregated news content at a conventionalweb page or web site as shown in FIG. 6. Furthermore the invention couldcomplement the prior art approaches. That is, a webpage or website couldbe configured so that the content is presented in the typical prior artstyle (i.e., based on the relevant factors identified by Google) alongwith the temporal ranked style described herein.

One other application of the invention is as an automated documentresearch/search engine. The above operations could be implemented in theform of a standalone software application or by a search engine thatassists in finding and compiling content related to topics of interestto a user. For example to research a story about a past event, a usercould specify different milestones in the form of textual expressions,such as: 1) Lincoln's birth; 2) Lincoln's childhood; 3) Lincoln's legalcareer; 4) Lincoln's legislative career; 5) Lincoln's presidency; 6)Lincoln's assassination, and so on. By using the milestones as a form ofreference set, the invention can locate appropriate content in thedesired temporal categories and match it to the user's desired temporalstructure.

Conversely a user may be allowed to present a free-form expression of adesired history of an event, such as by presenting open ended queriessuch as “Tell me about Lincoln's life” or “what important events havetaken place at this forum” and so on. The invention can be used tolocate applicable documents, sort them temporally, and present summariesof the various periods covered to allow the user finer control. Forexample the documents may be analyzed and grouped automatically indistinct time periods (covering certain decades, certain cultural era(hippies, wars)) and presented to the user in summary groupings. In theopen ended query example asking about Lincoln's life, the user may bepresented with the specific categories for Lincoln as noted above. Theuser could then drill down and study the individual temporal categoriesas desired. Other examples will be apparent to those skilled in the art.

With respect to step 170, again in some instances of course it may bedesirable to employ an algorithm which can perform a comprehensivescoring and sort in a single step. Thus either of the first or secondsteps could be omitted. Furthermore in some cases the seed set documentsmay consist of a single document for each category, in which case thecomparison to the reference document sets (and to other documents inthat category) may not be necessary.

A more detailed breakdown of a preferred document temporal scoringprocess 400 which can be used in step 170 (FIG. 1) is depicted in FIG.4. It will be understood that these are only representative of the typesof operations which could be implemented. As with the temporalizingprocedures noted above the operational steps of process 400 areimplementable as one or more software routines executing on a dataprocessing system that is coupled to the Internet.

At step 405 a source is identified for the document in question. Thisidentifier could take any form suitable for expressing the origin of thedocument, be it from a Blog, a news agency, an email, etc. The documentsources are catalogued and compiled for later reference. As notedherein, the sources are preferably correlated in a matrix/table (notshown) with the individual topics to identify a corresponding timelinessfactor. Furthermore the sources are also associated in some cases withspecific locales (see below) and such data can also be compiled in across reference table/matrix of any required form.

In some instances a source of a document may be difficult to attributebecause there is no identified author/source. One option that may beemployed in some instances is to use content/prose fingerprinting todetermine a correlation between the text of the document and a group ofauthors. By cataloging the idiosyncrasies, mannerisms, word choices,word frequencies, etc. of particular authors it is possible to devise adatabase of author/content characteristic pairings. That is, anauthor/source classifier can be trained from a corpus of materials inthe same manner as the document classifier above. In this manner, whenan unknown document is examined, a source classifier can analyze anddetermine a correlation and likely identity of a source/author of thedocument.

During step 410 the document is analyzed to determine one or moreappropriate topics associated therewith, in a manner akin to that whichwas described above for steps 150, 160 above in FIG. 1. Thus this can beaccomplished in any number of ways known in the art.

In step 415 the locale, situs or other regional specific informationabout the content or topic of the document is gleaned. For example inthe case of a sporting event, the situs would be the name of the citywhere it is taking place. This would be true as well for storiesinvolving natural catastrophes, accidents, disasters, etc. In some casesthe information can be extracted directly from the document but in otherinstances it may be necessary to incorporate other geo-locating routinesto place the locale of the event. This information is useful, asexplained further below, for identifying potential sources of currentinformation on the event in question as it is likely news reporters insuch regions will be most active in covering the story.

Step 420 processes the document further to catalog and strip down thecontent into manageable form. For example the title, identified topics,content, related image data and the like are catalogued and formattedinto appropriate form for use by later operational routines. At thistime a tentative temporal ranking field can also be appended if desired.

The next operation which takes place at step 425 is a determination of aclosest existing match to the document in question. This is done byreferring to a preexisting reference set identified as master topic set465 in FIG. 4. Again the process of determining such match can be donewith any number of techniques known in the art and implemented assoftware programs for comparing and matching content of electronicdocuments.

At step 430 the process determines if the document under evaluation ismerely a copy of an existing document. If so, then a notation is made tosee if it is from a new source or not. Should it be from a new sourcethe document is merely assigned the same temporal ranking as thepredecessor document from a prior source. The collection of sourceinformation is thus useful for identifying a relative timeliness ofspecific sources for particular topics, and can be used as explainedfurther below to enhance the efficiency of the system in locatingtemporally appropriate materials. If the document is instead merely arepeat from an already identified source it is simply discarded.

The ability of the present invention to quickly identify duplicates ishandy because in many cases it can be used as a first-pass filter foreliminating documents which would otherwise clutter a news aggregatorsuch as shown in FIG. 8. In other words in embodiments of the presentinvention the option to eliminate duplicates can result in a much moreefficient use of available window/page space.

The next step 440 (optionally) computes the actual differences incontent between the document and its closest match. The result of thisis as alluded to above is to determine which classification to assignthe document among the various options shown in FIG. 2. In other word,is the document A′, A″, A−B or A+B. This information can help to informand adjust the temporal rank to be given to the document. As with theother processes described herein, the process of determining differencesin content can be done with any number of techniques known in the artand implemented as software programs for comparing and matching contentof electronic documents.

Step 445 measures the temporal differences between the document and itsclosest reference match. This can be done in a number of different ways.First, as mentioned above, a document may have its content evaluated asa whole and classified with a temporal interpretation based on contenttags such as shown in FIG. 3. For documents in which the topic isrelatively well-known and the stories tend to be similarly behaving withtime, this is a relatively straightforward analysis. Thus the resultcould be that the document is given temporal ranking Tn. If Tn isgreater than Tref, the latter being the reference match temporalinterpretation, then the temporal rankings of the two documents areupdated at step 450.

Another variation of the invention could use a different analysis inwhich content snippets in the documents are determined and tracked fortemporal changes. This type of operation is shown with reference toFIGS. 4B, 4C and 4D.

In such type of approach content for a reference document 480 can beseen at the top of the figure. This content has a number of contentsnippets and tags which can be seen to provide useful guidance for laterupdates of the story in question, in this case an event in which a planehas gone missing. The content snippets 481, shown in boxes, are taggedautomatically by a natural language processor or manually by a humanoperator, depending on the particular application. A conventionaltext/word parser could be used to shred the documents into individualwords, phrases and sentences. It will be apparent that other contentstructures in the document, including images, graphics, audio data, andhyperlinks could be considered as well. Such items, along with anymetadata associated therewith, can also be examined to determine theirrelative age and timeliness.

This initial document, offering breaking news on a particular event,thus affords a baseline for later comparisons. For example, a snippetcan be comprised of a data entity of object/action pairs, preferably inthe form of {Object, Status} constructs. That is, in entry 480, someexamples could include {plane, lost contact); or {plane, missing} or{plane, efforts to contact} etc. Grammatically speaking, this analysiscould be considered a form of subject/predicate analysis. Since manynatural language engines are adapted to perform this type of linguisticanalysis of prose, this affords a relatively simple way to identifyitems with variable temporal related behavior.

The temporal related content snippets can be stored in an updateabletable which is constructed dynamically as documents are temporallydecoded. Thus it could be in the form:

TABLE 2 TEMPORALLY RELEVANT CONTENT SNIPPETS ID Object Status AgeRelated to Weight 1 Plane Missing 0 2 High 2 Plane Continuing 0 1 HighEfforts to contact

The age of the temporal content snippet can be used to determine arelevancy. That is, a snippet which is very old may be afforded lessweight because it is not matched with other temporal content insubsequent documents.

The “related to” field can be used to correlate snippets. For example, asnippet that describes 350 passengers could be related to another laterderived snippet which mentions 100 survivors, or another which describesa number of known fatalities. These relationships can be exploited toinfer/fill in missing information that might not be expresslyarticulated in later documents. Thus a certain amount of metadata canalso be constructed and organized for purposes of correlating temporalfeatures.

The weight factor is related to the value of the snippet as a temporalpredictor. This value is a function of how relevant a change in thestatus is relative to a real temporal change. In the example where anobject (plane) changes status from missing (one state) to found (anotherstate) the relationship is very strong and indicative of a temporaldifference between two documents. This weighting factor can be assignedany of a number of different values, either numeric or quantitative,depending on the application in question.

All of the factors above could be determined again automatically by acontent snippet routine, or by a human operator specifying the values.

Other formulations will be apparent to those skilled in the art andcould be used to dissect and characterize the content of the document ina temporal fashion for later comparison. The objects need not be nouns,as shown above, and in some instances may consist of phrases.

Entries 482, 483, 484 and 485, are later updates/reporting of the eventsof the main story 480. These documents (shown here for simplicity inedited form of course) can be analyzed to determine if they have tracesof the content snippets associated with the baseline document 480.Additional content snippets can be created as well to reflect newinformation gleaned in the updated stories.

Semantic supplementation can be employed (through the use of such toolsas WORDNET) to assist in the temporal decoding process. Consequently inthe present example, the term “plane” would be examined to determinesemantic equivalents, such as “aircraft” or “flight.” For largerlinguistic structures, such as phrases, the FRAMENET tool (or otherslike it) can be used to determine semantic equivalents. This facilitateslater comparison to other stories which use non-identical butsemantically related content.

Thus for entry 482, it can be seen that the {plane, status} contentsnippet has a counterpart {flight, gone down} which suggests that thelatter is more recent. Additional confirmation of the temporal changecan be found in the other snippets of entry 482, which notes for examplethat a rescue is already underway, since “efforts to contact” can againbe related to rescue. The latter snippet is particularly useful sincedisaster stories tend to fall along predictable story lines. Thus thepresence of such content in entry 428 can be correlated with a referenceset of disaster (or plane accident) related documents (see Table 1above) to identify that it is later in time than a document that merelyreports on a missing flight.

Entry 482 also shows additional snippets which are useful for latertemporal comparison. Namely, new information is now available on thenumber of people onboard, and the identity of one of them. These can nowbe the subject of new subject and predicate pairings. This could beclassified in the form of {carrying, 350 passengers} or {onboard, 350}or {passengers, 350}, and {Stink, onboard}. Consequently at the end ofprocessing the content temporal snippet table associated with thedocument for entry 482 table 2 would look like this:

ID Object Status Age Related to Weight 3 Plane Gone down 0 6 High 4Passengers 350 0 None Low 5 Stink Onboard 0 None Medium 6 Plane Rescue 03 Medium begun searching

In some cases it may be desirable to create a comprehensive master tablecombining entries from all entries. This can be used for determining anoverall status of each type of temporal content, by disambiguatingleaving single entries for each object. However for comparison purposesit may be useful to keep older metadata available for comparison.

From a comparison of the content snippet data for entry 482 and entry481, a determination would be made by a temporal routine that the formerrepresented content that is later in time. In a preferred approach thecontent snippets sharing common (and/or semantically related) objectsand status types would be evaluated between the two entries. In thiscase, the table entries for “plane” would be evaluated, and a temporaldecoder would determine that “gone down” (even qualified with “believeto have”) is reflective of a later temporal state than “missing.” Notethat it could be compared with a different status type as well—such as“efforts to contact” if desired and the result would be the same.

The table entries for #2 and #6 would be determined to be semanticallyrelated, by virtue of the fact that “rescue” and “efforts to contact”would be phrases denoting similar concepts. Thus an evaluation of suchsemantically related entries would also yield a result that the formeris a more recent term. Based on these evaluations the process woulddetermine that entry 482 should be ranked with a designation indicatinga more recent temporal value.

Similarly decoding of entry 483 reveals yet another content snippetrelated to plane (airplane) with the status “spotted”—which issemantically related to the concept of found. A new snippet is createdfor survivors in the form {survivors, looking for}. It should be notedthat other snippets could be coded of course, and in different forms;these are but examples highlighted to illustrate the important aspectsof the invention.

Entries 484, 485, 486 and 487 all show corresponding content snippetswhich can be matched to prior entry snippets. A comparison of some ofthe survivor content snips for example shows that a growing count can beattributed to a newer story. The update on Stink's status at entry 484is another example. It can be seen quite clearly that comparison oftemporal related snippets can yield reasonably reliable indications ofthe recency of documents, and the state of a particular event.

In some cases it will be apparent that there may be a many to one, orone to many relationship between temporal tags in different documents.This disparity in characterization of the state of the event can beexploited, of course, so that if one document only contains a single tagwith an object specifying a particular state, and a different documenthas several which identify the identical state, then the latter may beconsidered more reliable of course with respect to the temporal state ofthat particular event at least. So the frequency and absolute numbers ofobject/state tags can also be used of course to determine temporalorder.

FIG. 4C shows a similar evaluation of a story pertaining to a politicalelection. The entries 488-493 are shown in temporal sequence for ease ofunderstanding. It can be seen that by tracking content snippets 481describing the state of the polls {will open} {open} {officially over},precincts reporting {5%} {50%} {98%} {all} and vote tallies andpercentages the correct temporal sequence can be determined for a set ofstories.

Similarly FIG. 4D the evolution of a sporting event is shown with regardto a number of document entries 494-499. As with the other evolvingstories described above, the content snippets 481 can be used to trackthe state of various objects such as a time of the game {will kick off}{intermission} {3 minutes to go} {end} and the home visitor score {none}{0-7} {17-10} {27-24} {27-31}.

It will be understood again by those skilled in the art that otherconcepts/objects would be appropriate for other types of events.

Returning to FIG. 4 after the temporal ranking of the document iscompleted at 450, it is ranked and compared as noted in step 455. If thenew document has a higher temporal rating is then ranked higher than itsclosest counterpart then it is assigned a rank equal to such counterpartwhile the latter is demoted. Otherwise it is assigned a rankingequivalent to a document in the master set 465 with the same temporalrating. In some cases it may be desirable to fine tune a rating bycomparing such new document with the closest temporal match again byrepeating steps 440-445-450.

At step 460 the process then proceeds to select the next highest rankeddocument for comparison purposes with the new document. This nexthighest ranked document is then matched against the new document usingsteps 440-450 once again to determine a temporal winner. The process isrepeated until there are no further ranked documents to compare against(i.e., the new document is declared as the most recent document on theevent) or the new document fails to displace a prior ranked document. Inany event the new document, its temporal rating and its ranking are thenlisted in database 465. In addition a source field and databaseassociated with the document is updated to reflect the attribution ofthe temporal rating and ranking to a particular source. In this mannerthe behavior and performance of specific sources can be evaluated andrated over time as well on a topic by topic or story by story basis.

Other optional features which can be implemented in the above processinclude a verification step 480 which is explained in further detailwith reference to FIG. 5. This additional verification may be useful inthose instances where a document achieves a highest temporal rankingrating and a double check may be in order to provide additionalassurances of its temporal value.

Another optional feature is shown beginning with step 470 in FIG. 4. Inthis operation additional weighting/adjustments for the temporal ratingcan be given based on the value of the source providing the document.For example historical data 471 is consulted to identify prior temporalratings and rankings achieved by the source in question for thedocument. In addition a local relevance score is computed at 472 basedon a comparison of a situs of the event and a situs/associatedgeographical region of the source in question. For example, for a storyabout an accident in South Carolina, a station issuing reports fromCharleston would receive a higher rating than a similarly situatedstation in California reporting on the same event.

Accordingly at step 473 a temporal rating or a document ranking may beboosted or attenuated depending on the source timeliness evaluation. Itshould be noted that other factors may also be considered, for examplesources could also be measured and compared for their output on storieson particular topics. Over time a database or other correlation tablecould be developed that identifies sources known to be prolific onparticular subject matter areas (sports, finance, politics, culturalevents, entertainment, etc.) This prolificness factor, or fluency factormay be associated with particular interest or expertise of thereporters, writers or audience associated with the source, and can beused again to modify a temporal ranking score.

To facilitate source timeliness adjustments, it may be useful to compilea list of sources associated with particular geographic regions. Forexample all the TV stations and newspapers within a certain radius ofmajor cities, or within a radius of a set of coordinates. By identifyingand ranking a set of sources in advance—by examining historicalbehavior—the computation and adjustment can be expedited considerably.

Furthermore by identifying a situs of an event it may be beneficial tofocus attention and computing resources to monitor other geographicallysimilarly situated content sources as most likely originators of currentmaterial. Thus in the various operations noted above, such as steps 140,150 (FIG. 1) the search collection process could be focused and weightedmore heavily to local sources associated with the event. This isbecause, in large part, breaking stories in particular locales arelikely to be released to reporters/journalists which are well-known inthat community due to personal familiarity. Such local reporters arelikely tend to know their areas better, and are likely to know where togo to gather current information.

The ratings forum (discussed below) could be similarly tailored on ademographic/geographic basis so that content on particular topicsgermane to local events is presented to users in such area. Thisincreases the chances that persons with direct knowledge can participatein the authentication process.

A simple example of the value of the source timeliness factor can beseen in FIG. 4A. On this date (Aug. 4, 2007) at 8:40 PST, there is onlya single news story describing a situation in which a potential bombthreat was discovered in South Carolina. The news story was generated byWXLT, a TV station in South Carolina. From this and hundreds of otheravailable examples it is apparent that local news agencies can be aprime source of breaking news on local topics.

FIG. 5 illustrates an optional verification process 500 which can beused in conjunction with the aforementioned document temporalizer ofFIG. 1 and other related embodiments. The verification process, asmentioned above in connection with FIG. 4, can be used as a supplementalsupport mechanism to determine the relative recency or freshness of newcontent. The basic principle of this process is that theawareness/prevalence of certain content across a certain domain can bereflective of its recency. That is, with all other things being equal,if certain content on a topic which is believed to be relatively new canbe found in fact in only a fraction of certain benchmark sites, thenthis can be considered a reasonable indicator of its freshness.

At step 510 the new content of the document is extracted. For example,in a story about a sports game, the new content might include thecontent “ABC University has won the game against XYZ Tech.” In theverification process it may be desirable to use more than just the newtemporal related content to increase the query coverage.

During step 520 a set of verification sources is selected to serve asthe domain for the verification check. The verification sources, likethe local sources noted earlier, are preferably determined in advancebased on their historical performance as reliable indicators for thetopic in question. As an example, for a sports story dealing withfootball, it may be empirically determined that certain message boardposters on a social networking site are known to publish and postresults of sporting events very early compared to other sources.Consequently the verification sources can include searchengines/indices, other news outlets, RSS feeds, web logs (Blogs), socialnetworking site pages (including personal profiles, private messagewalls, etc.) message board sites, etc. Other examples will be apparentto those skilled in the art as they materialize on the Internet. Again,for reasons which are self-apparent, these same verification sourcesoften afford an excellent source for finding new materials as well foruse in the front end selection of the reference seed set (FIG. 1).

At step 530 an awareness or prevalence rate of the new content ismeasured in the domain. As an example, consider the story noted in FIG.4A discussed above. It can be seen in FIG. 5A that a search within theGoogle domain for the terms “Goose Creek Bomb” reveals a number of newpages created within a short period of time which appear relatively highon the search list. Thus it can be determined with some reasonablecertainty that the story is legitimate, it is timely, etc. Other sourcescould be explored as well.

If desired, an awareness or prevalence level or rating can be computedas well at step 540 for the new content. This can be used, again, todetermine if the content is authentic, reliable, or truly recent. Forexample, if the content elicits a very large number of hits, this couldindicate that the information is in fact alreadywell-known/disseminated, and thus actually stale. Conversely a smallnumber of hits within the domain might be construed as an extremely newstory based on the relatively small dissemination across the domain(s).

To improve accuracy the verification process could be executed atdifferent times to compare scores at different intervals. A story whichis rapidly changing in prevalence in a short period of time may beinferred to be relatively recent in many cases.

FIG. 6 illustrates a web page interface 600 that can support averification process as well (FIG. 1 reference 171 and FIG. 5). Theinterface is used to gather and present information to Internet usersconcerning news stories or other documents relating to particulartopics. Since it relies on participation by real users to rate/rankdocuments or stories it can be seen to be implementing a ratings boothfor persons to express their opinions. The ratings booth can beintegrated within a portal (such as the types offered by such companiesas Google, Yahoo!, Microsoft, AOL, etc.) or some other content providerwebsite (such as Motley Fool) or as an application that is used bymembers of a social networking site, or even provided as part of astandalone website which provides leading edge news.

Regions 610, 620 are used to present headlines of stories (A1, A2, . . .etc.) pertaining to particular topics. These stories are selected aspart of the top N list described above; while only 3 entries are shown,it will be apparent that any reasonable number of stories appropriatefor the interface can be presented. The interface collects data byallowing users to rank the stories relative to each other within regions610, 620 respectively. The documents are readable by the users by simplyselecting a URL embedded in the story field (not shown). A separateviewing window (not shown) could be used to peruse the story andcomprehend the content. The story headlines and a brief synopsis may bepresented within fields 630.

After viewing the same, users can indicate their opinion/belief on therelative recency of documents/news stories by ranking them in order fromtop to bottom. This can be achieved by simply dragging and dropping thestories in a particular order. For example the interface could requirethat users identify the most recent story by placing it at the top slot640. Alternatively a checkbox could be placed next to each to indicate arelative temporal rank, or a simple indication of the most recent one inthe group.

It is not necessary of course for users to rate all the stories. Simplyexpressing an opinion, however, concerning their impression of the mostrecent story can nonetheless be extremely valuable. The topics can ofcourse be customized and set up to particular user preferences. Forinstance a person interested in sports and finance could select suchtype of content to be presented within regions 610 and 620. Again whileonly two regions are shown it will be clear to skilled artisans thatmore space could be dedicated to such function.

Based on the user voting for the stories, a sorting routine (not shown)can then tabulate the data and present the results in region 660 forpublic viewing. The ranking data for this same list of stories can becommunicated by the sorting routine to the other routines used by theprocesses of FIG. 1 and FIG. 5 respectively above. The sorting routinecan be implemented in any number of ways appropriate to a particularenvironment by one ordinarily skilled in the art.

A separate reviewer tabulating routine (not shown) can then tabulate thedata concerning the most prolific reviewers and present the results inregion 670 for public viewing. This data is merely used to recognize thecontributions of reviewers who are assisting the ranking process, andcan be used to induce participation by users. Other inducements could beoffered as well depending on the nature of the particular portal/websiteimplementing the present invention, or the interests of an operator ofthe same.

Similarly, a separate reviewer accuracy tabulating routine (not shown)can then tabulate the data concerning the most accurate reviewers andpresent the results in region 680 for public viewing. As above, thisdata is used to recognize those reviewers whose contributions to theranking process are determined empirically to be most accurate from atemporal perspective.

The ranking data for this same list of stories can be communicated bythe sorting routine to the other routines used by the processes of FIG.1 and FIG. 5 respectively above Region 670 is reserved for presentingthe As with the sorting routine these tabulating routines can beimplemented in any number of ways appropriate to a particularenvironment by one ordinarily skilled in the art.

In some embodiments it may be desirable to bias and customize topicspresented to users based on their particular geographic area. Dedicatedareas may be used as well based on interface configurations selected bythe user. For example for a reviewer in Florida the Sports topicsstories may be more weighted on Florida based teams. This may increasethe enjoyment of the site for those viewers who prefer local stories. Inaddition the opportunity for local users to modify document rankings forstories in their respective geographic locales effectively converts suchpersons into informal reporters of sort who contribute to the accuracyand timely delivery of information for that region. In additionindividuals could be targeted based on their bookmarks, or queries madein the past, to determine their usefulness as validators for the topic,event or geographic region in question. In social networkingapplications, individuals could be solicited to become part of topicbased “reporting” groups to help contribute and refine content for thecommunity's benefit. Members can also subscribe to individual news“channels” on specific topics that are stocked in part with contentcontributed (or culled from other sources) by other members.

Other uses of the invention will be apparent to those skilled in theart. For example, one current line of research is heavily focused onpersonalizing search results for user queries. The ranked temporal listof stories by topic identified above in region 660 by the presentinvention could be used by search engine operators to modify and/orenhance search results. Consideration could also be made, of course, ofthe geographical source of the query as can be done by any number ofconventional search engine routines. For example a person entering aquery directed to certain key words such as “airplane accident” in thevicinity of the story discussed above in connection with FIG. 4B is morelikely interested in the local breaking story than on generalinformation relating to aviation safety. By consulting the most recenttemporal information relating to an event the relevance of searchresults is likely to increase as well as many people are reacting to therapid dissemination of information.

In similar fashion a recommendation engine could also be programmed toconsult with the temporal ranked list of topics/stories to render arecommendation to a user concerning an item, such as a book, atelevision program, a movie, etc. For example if a local story on anairplane crash is heavily followed and rated with a high temporalranking a television guide recommender may automatically record the nextavailable news report for a user. Similar recommendations can beaccommodated for other scenarios.

For certain embodiments of the invention it may be desirable to predictin advance the expected progress of a particular story. This predictioncan be used to form queries targeted in specific domains to identify andmine more content.

For example certain stories involving natural disasters, accidents,etc., tend to follow predictable patterns in terms of the evolution ofthe story. This basic pattern can be exploited. One example is anearthquake related event. Typically speaking the first stories reportsolely on the detection of the event. Then later there are usuallystories indicating the magnitude and epicenter. Still later come reportson damages, casualties, etc. From this known pattern a query formulationroutine can in effect predict or speculate on the nature of subsequentstories pertaining to a particular event. In other words, if a hot storyis found relating to an earthquake, a query formulation routine canconsult various reference sources (the USGS for example) to identify thelocation, for instance San Francisco Calif.

From there the query formulation engine could be begin to conductsearches in any desired domain (search engines, message boards, blogs,broadcasts, social network sites, etc) for keywords by incorporatingknown phrase terms such as “earthquake san Francisco {date}” andintegrating them with expected phrase terms such as “magnitude” or“epicenter” or “casualties” or “damage” etc. This is but an example ofcourse, and other stories would be predicted in accordance with atemplate/pattern developed for that type of event. As an example, for asporting event which is known to end at a certain time, a series ofsearches could be conducted with alternate predictions which cover thegamut of possible outcomes, such as “ABC University wins” or “ABC losesgame against XYZ” or “XYZ wins” etc., etc. This prediction technique hasthe advantage of exploiting the potential current coverage of a searchengine which can extract the data from unrelated sources in a moretimely fashion than a dedicated software robot which is searching andcompiling data from multiple sites.

As noted above many natural disaster stories can be anticipated to someextent by their very nature. Thus for an earthquake centered in aparticular area, it may make more sense to tap into content sourceslocal to such event. In the case of a hurricane, a weather center couldbe consulted to determine an expected path of the storm. This pathinformation could be used again to target local media reporting sourcesto increase the relevance and timeliness of information.

For other planned events, such as sports, political summits,entertainment related, etc., similar types of predicted news storiescould be tested to identify actual relevant content as it is just cominginto existence.

A further source of information that can be exploited are closecaptioned television broadcasts, podcasts, RSS feeds and similar datafeeds which can include text (or audio that can be recognized). Thesecan afford additional opportunities for recovering timely data.

The invention could also be used in connection with systems whichmonitor activity in message boards, Blogs, RSS feeds, social networkingsites, etc. In such instances the invention can be assisted by a pollingroutine which operates periodically to scan and retrieve content fromparticular designated sites, in a manner similar to that shown anddisclosed in U.S. Pat. No. 6,493,703 incorporated by reference herein.This polling can be done on a topic or event basis, and suchtopics/events can be determined automatically based on the content of aweb page/site in question (in a manner similar to that done byimplementations of Ad Sense™—a Google ad serving technique). Forexample, a fan site dedicated to the Boston Red Sox baseball team mayhave a dynamic list of message board posts, blogs, etc., with currentnews on the team, specific players medical conditions, scores, etc. Thesame technique could be incorporated in message board systems devoted toparticular equities; that is, a list of top stories reflecting thestatus of some company event (earnings, product releases, court results)could be maintained for the pleasure of the board participants. A socialnetworking site could have programmable pages, so that users could electto designate certain areas of interest which they would like to seeupdated periodically to reflect the most current state of knowledge ofthe social networking site itself, or beyond such domain if required.The advantage of the invention is that it becomes possible now toidentify particular third party sources that are most apt to producetimely content on particular topics/events such as bloggers.

Alternatively the topics/events could be explicitly specified by avisitor to the web page in question. In the above example a user couldtype in a search pertaining to a particular player's status, and thepresent invention would poll a specified target list of sources todetermine a best answer to the user's query. The frequency, sources,topics, etc., could all be programmed based on the nature of the contentserved by the site/page in question.

In a social networking application, the individual personalized pages ona website could be examined. One possibility that has not yet beenexploited is cell phone and other audio based communications betweenindividuals. In a collaborative environment some individuals may chooseto permit broadcasting and/or monitoring of their communications,including text messages, for public consumption. The motivations fordoing so will vary in accordance with each application, but againsuitable inducements may exist for allowing such types of ecouterisms.By tapping into such communications, and decoding them into textstreams, it is possible again to derive another source of event relateddata.

Search engine results could also be modified in accordance with atemporal characteristic. It is known in such field of art to try anddetermine the time/age of particular pages through explicit time stampsto help prioritize and provide a different look for search results. Aswith publication/release times for articles, time stamps do notnecessarily reflect current conditions of an event. The same is truewith respect to message board systems, where people frequently repeatold content and new content becomes buried under an avalanche of oldinformation. The present invention could be used in lieu of or inaddition to such techniques, so that the search results are parsed andanalyzed to determine their relative temporal order relative to atopic/event determined by the search engine from the user's specifiedquery text (“who won the TX election”, “what were X's earnings”) orrelative to a topic/event gleaned by the present invention whenexamining the actual search engine results.

As seen in FIG. 10 an electronic advertising process 1000 can also beaffected and modified by temporal parameters of documents. For examplean online content presenter may note that a user is reviewing a set ofdocuments on a topic in a particular temporal order. The advertisingpresented to such user can be selected/adjusted in response todetermining a temporal parameter of a document being presented.

For example at 1010 a user reads that the status of a hockey game is atan intermission with his favorite team winning. This temporal (andcurrent outcome) state of the event can be used to influence the type ofadvertising presented to the user at 1050, since individuals tend tohave particular psychological associations with sporting event breaks,team successes, etc. and other products Many alcoholic beveragecompanies tend to present television advertising during such occasions.Other examples will be apparent to those skilled in the art based onbasic market research, and based on an expectation of a user's mentalstate/demeanor during review of content.

Furthermore (and as a complementary approach) it may be determined thatan event has terminated, but that the user has not completely reviewingcontent on a particular matter as shown at step 1020. Since it can beexpected that the user will eventually discontinue their browsing on thesubject upon reaching the end of a temporal state for the event, theauction algorithm for an advertising engine can be optimized at 1040 toadjust pricing of keywords/ads to such individual. This same principlecan be applied at any stage of the user's review of content todynamically price the cost of advertising/keywords.

Thus prices for ads presented early on in a session can be priceddifferently than ads presented later in a session, based on anadvertiser/event state database 1030 for ads/keywords. This is similarto pricing models used by television advertisers, who typically receivedifferent price points for content presented at different stages of anevent. Because the invention can be used to determine a temporal reviewstate by the user (including a relative temporal order relative to afinal state, the rate of consumption of material by the user and theamount of material still left to be reviewed) a prediction can be madeof the user's expected overall session time on such topic. By adjustingan advertising price in accordance with a user's expected session time,and on a topic by topic basis, the invention can improve advertisingeffectiveness, budgeting, etc. Accordingly advertisers can be presentedwith options/keyword variations for presenting ads at different temporalsession states (when the user is reviewing content for a completedevent), or based on certain event states (i.e., at the beginning of agame, as compared to an end of a game).

Again this same principle can be applied to search engine behavior aswell, so that queries for a topic are processed in part based ondetermining a current state of awareness by a user of content for anevent, and attempting to present him/her with more relevant informationbased on identifying such state. Therefore an individual making a queryat time T for an event, and presented with a certain amount of content,can be expected at a later time T1 to have an awareness of a state ofthe event from the earlier time. For such person it may make sense insome cases (or at the user's option) to only present information for themore updated state of the event as seen at 1050.

Finally, it will be apparent to those skilled in the art that themethods of the present invention, including those illustrated in theabove figures can be implemented using any one of many known programminglanguages suitable for creating applications that can run on clientsystems, and large scale computing systems, including servers connectedto a network (such as the Internet). Such applications be then beembodied in tangible, machine readable form for causing a computingsystem to execute appropriate operations in accordance with the presentteachings. The details of the specific implementation of the presentinvention will vary depending on the programming language(s) used toembody the above principles, and are not material to an understanding ofthe present invention.

The above descriptions are intended as merely illustrative embodimentsof the proposed inventions. It is understood that the protectionafforded the present invention also comprehends and extends toembodiments different from those above, but which fall within the scopeof the present claims.

1-27. (canceled)
 28. A method of verifying content of a news story witha computing system comprising: a) monitoring published items with thecomputing system in a knowledge domain including one or more socialnetwork domains, news content domains, blog domains and/or message boarddomains; b) processing news content with the computing system in a firstset of items published in said knowledge domain to extract at least afirst set of events referenced in such news content; c) identifying atleast a first event in said first set of events described in a firstcontent item published in said knowledge domain; d) automaticallyverifying with the computing system if said first event in said firstcontent item has occurred in fact; wherein said automatically verifyingis performed automatically in real-time by checking one or moreverification sources; e) generating a verification rating for said firstevent indicating a result of said verification step (d).
 29. The methodof claim 28 wherein an automated check is performed during step (d) todetermine if a news story identified in said first content item islegitimate.
 30. The method of claim 28 wherein a news story isidentified in said first content item and a timeliness of said firstevent relative to other events for said news story is automaticallydetermined.
 31. The method of claim 28, further including a step:presenting said first content item as part of a news feed only when saidverification rating exceeds a predetermined threshold.
 32. The method ofclaim 28 further including a step: automatically determining names ofpersons in said first content item.
 33. The method of claim 32 furtherincluding a step: determining a time based sequence of events forpersons in said news content based on temporal decoding of said newscontent other than timestamps for such first set of items.
 34. Themethod of 28 wherein step (d) is performed by automatically reviewingverification sources in said knowledge domain.
 35. The method of claim28 wherein step (d) is performed by automatically reviewing verificationsources separate and outside said knowledge domain.
 36. The method ofclaim 28 wherein said verification sources are processed and given areliability rating for accuracy, and are selected at step (d) based onsuch reliability rating.
 37. The method of claim 28 wherein saidverification rating is provided in party by selected human sources thatare also consulted during step (d).
 38. The method of claim 36 whereinsaid selected human sources are determined by automatically detecting anevent site identified in said first content item, and such that onlyselected human sources proximate to event site are consulted.
 39. Themethod of claim 38 wherein speech input from said selected human sourcesis recognized and decoded for said verification.
 40. The method of claim38 wherein fixed length limited text messages provided by said selectedhuman sources is used for said verification.
 41. The method of claim 28wherein step (d) is performed periodically by the computing system todetermine a change in prevalence in said verification sources.
 42. Themethod of claim 28 further including a step: selecting and presenting anadvertisement to a user reviewing said news content in an electronicinterface, which advertisement is based on first content automaticallydetected for said first item and a site of said first eventautomatically detected.
 43. The method of claim 28 further including astep: selecting and presenting an advertisement to a user reviewing saidnews content in an electronic interface, which advertisement is based onfirst content automatically detected for said first item and aprediction of a mood for said user also determined automatically fromsaid first content.
 44. The method of claim 41 wherein said moodprediction is also based on characteristics of the user, including anidentified affinity with entities identified in said first content. 45.The method of claim 28 further including a step: predicting futurecontent of future unpublished stories in said knowledge domain based onautomatically processing said first event.
 46. The method of claim 43wherein said future content is determined by automatically identifyingprior news stories containing said first event or similar events andfurther temporal sequenced content relating to later developments ofsuch news stories.
 47. The method of claim 43 wherein said futurecontent is determined by a parser classifying said first event, and byreferencing a template specifying a predetermined predictable change incontent for said first event.
 48. The method of claim 28 wherein saidnews content is in a news broadcast.
 49. The method of claim 28 whereinsaid automatically verifying with the computing system is performed byconstructing and retrieving results from an automated query based onfirst content keywords extracted from said first content item.
 50. Themethod of claim 49 wherein said automated query also includes predictedkeywords determined from phrases determined to be correlated to saidfirst content keywords.
 51. The method of claim 49 wherein saidautomated query also includes predicted keywords determined from phrasesdetermined to be temporally related to said first content keywords. 52.The method of claim 49 wherein said automated query is repeated laterwith a second set of verification sources predicted to be relevant basedon a progress predicted for said first set of events.